Method of hyperspectral measurement

ABSTRACT

The present disclosure provides a method of providing a hyperspectral image of a scene comprising a point (p). The method comprises providing an imaging device ( 10, 10 ′) comprising a two-dimensional image sensing unit ( 12, 13, 14 ) having a spectral characteristic, which varies along at least one direction in a plane parallel with an image sensor ( 13 ), acquiring a first two-dimensional image ( 11 ) of the scene, the first image comprising the point (p), wherein the spectral content of the scene varies along the direction as a consequence of the varying spectral characteristic of the image sensing unit ( 12, 13, 14 ), moving the imaging device ( 10, 10 ′) and the scene relative to each other, acquiring a second two-dimensional image (I 2 ) of the scene, the second image comprising the point (p), wherein the spectral content of the scene varies along the direction as a consequence of the varying spectral characteristic of the image sensing unit ( 12, 13, 14 ), identifying the point (p) in the second image (I 2 ) as a second pixel at a position (x 2 , y 2 ) depicting the point (p), providing a spectral vector (S(p)) of the point, and providing an updated spectral vector (S′(p)) of the point based on an applicability vector (A(p)) of the point (p), the spectral vector (S(p)) of the point, a spectral value (z 2 (p)) of the second pixel and an applicability vector (B(p)) of the second pixel.

TECHNICAL FIELD

The present disclosure relates to a method of hyperspectral measurement,which can be used also for providing three dimensional mapping data.

BACKGROUND

In both military and civilian remote sensing the use of hyperspectralcameras, that is, cameras that can measure received light in manywavelengths simultaneously, is increasing. In a hyperspectral camera,the number of observed wavelengths is dozens or hundreds, not just red,green and blue like a common color camera. Thus, each pixel contains theobserved spectral signature of an object or a point on the ground, i.e.,a set of values, one for each observed wavelength. After compensatingfor atmospheric effects, sunlight, etc., the spectral signature can beused to determine the material of each pixel. This can be used to, forexample, distinguish between different types or conditions of vegetation(environmental monitoring, precision agriculture) or distinguish betweennatural and artificial greenery (find camouflage nets hidden in nature).

At the same time, the need for 3D mapping has increased, and it istherefore common to complement hyperspectral airborne mapping with alaser scanner. Combining data from the two sensors, a three-dimensionalhyperspectral map is created. In recent years, methods for so-calledpassive 3D, which uses “standard” cameras for estimating 3D structure,have been introduced. However, such methods cannot be used with existinghyperspectral cameras, as these observe one line at the ground, not anentire image. Thus they do not see a point on the ground from severalangles, which is required for the estimate of the 3D structure.

WO2014/140189A2 discloses a method of providing hyperspectral images

SUMMARY

An object of the present disclosure is to provide methods and deviceswhich improve the availability of hyperspectral measurement and/or 3Dmapping, e.g. by reducing weight, cost of manufacturing and/or need forprocessing power.

According to a first aspect, there is provided a method of providing ahyperspectral image of a scene comprising a point. The method comprisesproviding an imaging device comprising a two-dimensional image sensingunit having a spectral characteristic, which varies along at least onedirection in a plane parallel with an image sensor, acquiring a firsttwo-dimensional image of the scene, the first image comprising thepoint, wherein the spectral content of the scene varies along thedirection as a consequence of the varying spectral characteristic of theimage sensing unit, moving the imaging device and the scene relative toeach other, acquiring a second two-dimensional image of the scene, thesecond image comprising the point, wherein the spectral content of thescene varies along the direction as a consequence of the varyingspectral characteristic of the image sensing unit, identifying the pointin the second image as a second pixel at a position depicting the point,providing a spectral vector S(p) of the point, and providing an updatedspectral vector S′(p) of the point based on an applicability vector A(p)of the point, the spectral vector S(p) of the point, a spectral value ofthe second pixel and an applicability vector B(p) of the second pixel.

The image may be a hyperspectral image, i.e. an image, which for eachpixel of the image provides spectral information of four or morewavelengths, preferably five or more or ten or more, e.g. 5-150wavelengths.

The imaging device may present a spectral characteristic which varies,that is, different parts of the image sensor record light at differentwavelengths. In consequence, the spectral content may vary over anyimage produced by the imaging device.

A two-dimensional image may be defined as an image that actually depictsthe scene. That is, the image may be a true depiction of the scene,apart from the fact that the spectral content varies along thedirection.

The relative movement may preferably be along a direction which isparallel with the direction along which the spectral characteristicvaries.

The number of wavelengths provided for by the applicability vector maybe typically smaller than the number of wavelengths and/or pixelsavailable from the image sensing unit along the direction. Typically,the number of wavelengths may be 5-150, 5-100, 5-50 or 10-30.

The elements of the applicability vector A(p) may be normalized values.The applicability vector A(p) may also be an index vector, e.g., A(p)=[0. . . 0, 1, 0 . . . 0] to indicate the specific wavelength of theposition (x, y) of the point (p) in an image.

The elements of the applicability vector B(p) may be normalized values.The applicability vector B(p) may also be an index vector, e.g., B(p)=[0. . . 0, 1, 0 . . . 0] to indicate the specific wavelength of theposition of the pixel depicted the point in an image.

By using this method, a spectral vector of each point may be providedthat contains sufficient information for it to be useful, but which isstill of a manageable size. Thus processing and storage requirements maybe mitigated and consequently cost and size of the imaging device.

The method may further comprise providing the applicability vector A(p)of the point to describe, with respect to predetermined wavelengths, thespectral characteristic of the image sensing unit at a position wherethe point is depicted.

An applicability vector A(p) may be provided for each point p and isupdated as more data on the point is received.

The applicability vector A(p) may provide a measure of the reliabilityof measurement at each point to control the weight given to the spectralvector S(p) when new measured values are introduced. The applicabilityvector A(p) may also be an index vector, e.g., A(p)=[0 . . . 0, 1, 1, 1,0 . . . 0] to indicate the specific wavelengths of the measured value ofthe point (p) in an image.

The method may further comprise providing the applicability vector B(p)of the second pixel, said applicability vector B(p) describing, withrespect to a predetermined wavelength, the spectral characteristic ofthe image sensing unit at the second pixel.

A new applicability vector B(p) may be provided for each pixel and foreach new second image.

This applicability vector B(p) may provide a measure of the reliabilityof the measurement at each pixel to control the weight given to themeasured value z(p) from the second image when incorporated into thespectral vector S(p).

The method may further comprise providing an updated applicabilityvector A′(p) based on the applicability vector A(p) of the point and theapplicability vector B(p) of the second pixel.

An updated applicability vector A′(p) may be provided as a maximum, onan element by element basis, of the applicability vector A(p) of thepoint and the applicability vector B(p) of the second pixel.

For example, each element A_(i)′(p) of the updated applicability vectorA′(p) may be provided as max(A_(i)(p), B_(i)(p)).

Identifying the point may comprise using the spectral vector S(p) of thepoint.

The identifying may comprise identifying the point based on a differencebetween the second spectral value and the spectral vector S(p).

For example, the point may be identified based on a minimal differencerelative to the spectral vector.

The identifying may comprise identifying the point based on the spectralvector (S(p)), the second spectral value, the applicability vector A(p)of the point and the applicability vector B(p) of the second pixel.

For example, a match may be provided based onnorm((S(p)−z2(p))·*A(p)·*B(p)).

The identifying the point in the second image may comprise providing amodel of a neighborhood of the point in one of the images, and matchingthe model to the one of the images.

The method may further comprise determining a respective observationangle of the point based on respective positions of the point (p) in thefirst and second images and a reference distance, and determining adistance between the imaging device and the point based on the anglesand the reference distance.

A reference distance may be a distance between the positions of theimaging device when acquiring the images, or a known distance betweenthe imaging device and a part of the scene, or between known parts ofthe scene.

Providing a spectral vector S(p) of the point may comprise assigning afirst spectral value of a first pixel depicting the point in the firstimage.

Providing a spectral vector S(p) of the point may comprise receiving thespectral vector S(p) already updated based on the first image.

The varying spectral characteristic may be provided by a variableoptical filter arranged adjacent the image sensor and the method mayfurther comprise providing the applicability vector A_(i)(p) of thepoint based on a transmittance of the optical filter at each wavelengthat the respective position of the image sensor where the point isidentified.

A “variable optical filter” may be defined as a filter, which only letsthrough the light of a specific wavelength, but which wavelength variesin at least one direction in the plane of the filter. The variation maybe in a first direction of the filter and the filter characteristic maybe constant in a direction perpendicular to the first direction. Thefilter characteristic should be known at any part of the filter. Forexample, the filter characteristic may vary linearly along the firstdirection. Also, the filter may be continuous along the first direction,such that longer wavelengths are transmitted at one edge of the filterand shorter wavelengths at the opposite edge of the filter.

The variable optical filter may be arranged in an optical path between alens and the image sensor. Typically, the variable optical filter may beprovided immediately adjacent the image sensor.

The variable optical filter may cover the entire effective surface ofthe image sensor. That is, all light that is recorded by the imagesensor will have passed through the filter.

For example, the applicability vector A(p) of the point p may beprovided as A_(i)(p)=τ_(xy)(λ_(i)) for each wavelength, wherein τ_(xy)is the transmittance of the filter at position x, y and i=1 . . . N.

An element A_(i)(p) of the applicability vector A(p) of the point p maybe a normalized value of τ_(xy)(λ_(i)).

The filter may vary continuously along the direction, preferably over atleast 90% of a sensor length in the direction.

The filter may present at least two spaced-apart portions having thesame transmittance.

The spectral vector S(p) of the point may be provided as anN-dimensional vector comprising a predetermined number of wavelengths λ₁. . . λ_(N), wherein N preferably is 5-150.

A respective spectral vector S(p) may be provided for a predeterminednumber of points in the scene, said predetermined number of points beingsufficient to positively identify an object or absence of an object inthe scene.

Acquiring the first and/or second image may comprise acquiring an imagewherein the point occurs only once.

Acquiring the first and/or second image may comprise acquiring an imagewherein the point occurs at least twice.

The method may further comprise determining a respective observationangle of the points (p) based on respective positions of the occurrencesof the point (p) in the first and/or second images and a distance whichis an intrinsic distance of the imaging device.

The intrinsic distance may be e.g. a distance between the portions ofthe image sensor where the occurrences of the point are recorded, adistance between focal points of lenses or a distance between lens andimage sensor.

The imaging device used in the method may comprise an optical variablebandpass filter.

The method may comprise estimating, e.g. interpolating, at least oneelement of the spectral vector, which element corresponds to awavelength for which measurement has been made with respect to thepoint.

The method may further comprise, if the point has moved more than onecolumn on the image sensing unit between the first and second images,then estimating, e.g. interpolating, elements of the spectral vectorcorresponding to wavelengths where no measurement has been done.

The estimation (e.g. interpolation) may be performed based on theapplicability vector.

For example, the value of the element(s) that is(are) being estimatedmay be determined partially by adjacent values and partially by thevalue of the applicability vector of the point, for example as aweighting factor.

Alternatively, or additionally, the applicability vector may be updatedbased on the element that has been estimated, e.g. in the same manner asdisclosed with respect to the formation of the 2D hyperspectral image.

According to a second aspect, there is provided a method of providing ahyperspectral 3D representation of an object, comprising providing aplurality of spectrally varying images depicting the object, providing a3D reconstruction of the object based on the plurality of spectrallyvarying images, and assigning a hyperspectral signature to a pluralityof 3D points on a surface of the object.

A “spectrally varying image” is an image, i.e. a two dimensionalrepresentation, wherein the spectral content of the object depictedvaries along at least one direction as a consequence of the varyingspectral characteristic of the image sensing unit.

The spectrally varying images may be received online, essentially inreal time as they are captured, or they may be received from a storagemedium for use in a post processing step.

In this method, providing a 3D reconstruction of the object may compriseproviding a set of intrinsic parameters for the camera, such as focallength, lens distortion, etc., providing a set of extrinsic parametersfor the camera, such as camera position and orientation, and providing aset of 3D points on the surface of at least part of the object.

A 3D point may be defined as a set of coordinates describing theposition of the point.

The method may further comprise providing a 3D surface polygon model ofat least part of the object.

The method may further comprise, for at least some of the spectrallyvarying images, projecting the at least some of the 3D points onto animage plane of the spectrally varying image based on the extrinsic andintrinsic parameters, to determine respective image specific coordinatescorresponding to said 3D points.

“Image specific coordinates” are coordinates which are specific to onespectrally varying image, and which describe the location of a specific3D point in that specific spectrally varying image.

The method may further comprise traversing at least some of the cameraparameters for at least some of the reconstructed 3D points, identifyinga matching 3D point were the projection of the 3D point on thespectrally varying image results in valid image coordinates, andassociating an index of the spectrally varying image with coordinates ofthat 3D point.

Preferably, all intrinsic and extrinsic camera parameters may betraversed for each reconstructed 3D point.

The method may further comprise reconstructing a hyperspectral signatureof at least some of the 3D points by traversing the associations ofspectrally varying image indexes for the 3D point to provide a spectralvector for the 3D point.

The method may further comprise estimating, e.g. interpolating, elementsof the spectral vector corresponding to wavelengths where no measurementcan be found among the spectrally varying images.

In the method, assigning a hyperspectral signature to one of the 3Dpoints may comprise identifying the 3D point in a first one of thespectrally varying images, identifying the 3D point in a second one ofthe spectrally varying images as a second pixel at a position depictingthe 3D point, providing a spectral vector of the 3D point, providing anupdated spectral vector of the 3D point based on an applicability vectorof the 3D point, the spectral vector of the 3D point, a spectral valueof the second pixel and an applicability vector of the second pixel.

According to a third aspect, there is provided an imaging device,comprising a two-dimensional image sensor, presenting first and secondmutually orthogonal directions, a optical filter, arranged in an opticalpath to the image sensor, wherein the optical filter is an opticalbandpass filter, the transmittance of which varies along on the firstdirection and being constant along the second direction, wherein theoptical filter extends over at least 90%, preferably 95% or preferably99% of a length of the image sensor in the first and second directions,respectively, and wherein the imaging device is arranged to acquiretwo-dimensional images of a scene, wherein the spectral content of thescene as depicted on the images varies along the first direction as aconsequence of the varying transmittance of the optical filter.

The optical filter may be continuous along the first direction.

The filter may thus vary linearly or non-linearly. Typically, such afilter does not contain any two portions (spaced apart in the firstdirection, which have the same transmittance.

The optical filter may present at least two spaced-apart portions havingthe same transmittance.

The imaging device may further comprise a lens, which is arranged in anoptical path towards the image sensor, wherein said lens provides onefocal point.

The imaging device may further comprise a lens or lens set, which isarranged in an optical path towards the image sensor, wherein said lensor lens set provides at least two focal points.

The imaging device may further comprise a processing device, which isconfigured to perform the method as disclosed above.

The present disclosure provides a new device and a method that togethersolve several problems with existing technology. The new device is basedon mounting a newly developed optical filter on top of the sensor chipin a digital camera. The filter only lets through the light of aspecific wavelength, but which wavelength varies linearly along thefilter. Thus, in the resulting image, pixels along one edge of the imageshow the amount of light with long wavelength (e.g., red light), andpixels along the opposite edge show the amount of light with shortwavelength (e.g., blue light). If the camera is mounted in an aircraftand moves over an area, the light from a point on the ground is firstmeasured at one wavelength, and as the object moves in the image, itwill be measured at different wavelengths. When the ground pointeventually passes out of the field of view, two things have beenachieved: The incoming light from the ground point has been measured indifferent wavelengths, and it has also been measured from differentangles. In principle, the 3D position of the point as well as itsspectral signature can then be computed, as detailed below.

The proposed device has several benefits over existing technology: Theweight is low compared to existing hyperspectral cameras, enabling thedevice to be carried by small electrical drones; commercially availablecomponents can be used, keeping the price down as well as allowing ahigh spatial resolution; simultaneous measurement of spectral signatureand 3D structure can be done.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates an imaging system 10, 20.

FIG. 2 schematically illustrates an image sensing unit.

FIGS. 3a-3c schematically illustrate parts of the image sensing unit.

FIG. 4 schematically illustrates the camera 10, 10′ at two differentpositions.

FIGS. 5a-5b schematically illustrate the parts of the image sensingunit.

FIGS. 6a-6c schematically illustrate the derivation of a part of aspectral vector S(p).

FIG. 7 schematically illustrates the spectral vector S(p) as arepresentation of the actual spectrum.

FIGS. 8a-8c schematically illustrate the updating of a spectral vectorS(p).

FIG. 9 schematically illustrates an image sensing unit comprising morethan one lens.

DETAILED DESCRIPTION

Referring to FIG. 1, the device may be embodied as an electro-opticalsensor 10 consisting of a sensor chip with a focal plane array (FPA) 13and read-out electronics 14, an optical filter 12, and optics 11, suchas a lens, see FIGS. 1 and 2. The sensor chip and optics can be standardcomponents from existing cameras, such as high-end DSLRs or machinevision cameras. The optical filter may be attached on the FPA, or in thedirect proximity to of the FPA, or it may be manufactured as a layer ofthe FPA.

Referring to FIG. 9, the device may comprise a lens set 11′, which isarranged in an optical path towards an optical filter 12′, a focal planearray (FPA) 13′ and read-out electronics 14′.

The lens set 11′ may comprise at least two lenses to provide at leasttwo focal points. The lenses may be arranged to immediately adjacenteach other e.g. in a row, in a column or in other patterns, or bearranged spaced from each other e.g. in a row, in a column or in otherpatterns.

The lens set may be formed as separate lens parts, which are mounted inthe desired pattern. Alternatively, the lens set may be formed as one ormore components, each of which making up two or more effective lensportions.

The lenses or lens portions may be formed as traditional lenses or asany other type of structure providing the function of a lens, such as adiffractive optical element.

The lens set 11′ may comprise one lens providing at least two focalpoints.

The optical filter may be a bandpass filter letting light pass only in anarrow wavelength band, centered at a wavelength λ_(c). This centerwavelength varies over the filter, so that the center wavelength is afunction of the position x, y, see FIG. 3, that is λ_(c)=λ_(c)(x, y).Such a filter is here called an optical variable bandpass filter (OVBF).In the preferred implementation, the center wavelength variescontinuously along one of the dimensions (here called x) of the filter,so that the center frequency is a function of x only, that isλ_(c)=λ_(c)(x). Also, in the preferred implementation, λ_(c)(x) is alinear function, and the filter is then said to be a linearly variablefilter (LVF). However, for the purposes of this invention, the filter isneither required to be linear or one-dimensional as long as it is knownand preferably continuous.

As the OVBF is mounted on, or very close to, the FPA, the lightregistered by a sensor element at position x, y will only containwavelengths close to λ_(c)(x, y), as illustrated in FIG. 3. When thesensor and the observed surface 30 are static, each point on the surfacewill thus be observed in a specific wavelength.

When there is relative motion between the sensor and the surface, theprojection by the optics of a certain point p on the object will moveacross the FPA. This is illustrated in FIG. 4, where a point 31 is firstobserved by the sensor 10 at one position and then by the sensor 10′ atanother position. When the sensor at the first position 10 observes thepoint 31, it will be projected on a sensor element 15 at position x₁, y₁on the FPA and the light is filtered by the OVBF 12 at the same positionx₁, y₁. When the sensor at the second position 10′ observes the point31, it will be projected on a sensor element 16 at position x₂, y₂ onthe FPA and the light is filtered by the OVBF 12 at position x₂, y₂.

When the lens or the lens set provides at least two focal points, theprojection by the lens or the lens set of a point p on the object mayprovide at least two different images of the object comprising the pointp on different positions of the FPA. This is illustrated in FIG. 9.

When a point p is thus observed at by different sensor elements 15, 16,a spectral signature X(p) can be estimated for each such point p. For anLVF, the vector S(p) has the same number of elements as the number ofsensor elements along the x-direction on the FPA, so that a measurementon the i:th column on the FPA corresponds to the i:th value in thevector S(p). Moreover, if the projection of a point p moves more thanone column on the FPA between two observations, there will be gaps,i.e., the missing elements of the spectral vector S(p) corresponding tothe wavelengths where no measurement is done). The applicability vectorA(p) may be used to interpolate the missing elements such that the gapsof the spectral vector S(p) can be removed.

For a high resolution FPA, with thousands of sensor elements along eachcolumn, this requires large amounts of memory and processing power, asthe vector S(p) would contain thousands of values for each tracked pointp. While such an embodiment is technically feasible, its practicalusefulness may be limited by factors such as the available memory andprocessing capacity.

The OVBF is at each point (x, y) characterized by its spectraltransmittance τ_(x,y)(λ), that is the amount of the incoming light ateach wavelength A that passes through the filter. The spectral radianceof the light that has passed the filter at point (x, y) isL_(out)(λ)=τ_(x,y)(λ)*L_(in)(λ), where L_(in)(λ) is the spectralradiance of the incoming light, as illustrated in FIG. 5. By definitionof a bandpass filter, the transmittance function τ_(x,y)(λ) is centeredaround λ_(c)(x, y).

In the present invention, the above-mentioned measurements of lightradiating from a point p at different wavelengths (corresponding todifferent positions on the FPA) are collected into an N-dimensionalspectral vector S. The values of S are estimates of L_(in)(λ) for anumber (N) of wavelengths λ₁ . . . λ_(N), where N is the dimensionalityof S. N can be chosen arbitrarily considering the needs of theapplication and the amount of available memory and computational power.Typical values of N may be 5-50, preferably 5-25, 5-15 or 5-10.

When the first measurement z of a point p is made, at position (x, y) onthe FPA, the values of S(p) are set to the retrieved measurement valuez. In addition, an applicability vector A(p) for the point p is created.The applicability vector takes its values A_(i)(p) from the knowntransmittance of the filter at each wavelength, that isA_(i)(p)=τ_(x,y)(λi) for i=1 . . . N.

Each time an additional measurement is made at point p, the measurementgives a new measurement value z(p) and an associated applicabilityvector B(p). The spectral signature S(p) is then updated taking intoaccount the new measurement as well as the old and the new applicabilityvectors. In its simplest form, this update is done by setting each valueS_(i)(p) in the spectral signature S(p) to

S_(i)(p)*A_(i)(p)+z(p)*B_(i)(p).

FIG. 8a illustrates the current spectral signature S corresponding tothe spectral vector S(p) and the updated spectral signature Scorresponding to the updated spectral vector S′(p), FIG. 8b illustratesthe current applicability function A corresponding to the applicabilityvector A(p) and the updated applicability function A′ corresponding tothe updated applicability vector A′(p). FIG. 8c illustrates theapplicability function B corresponding to the applicability vector B(p)used to update the spectral vector S(p) and the applicability vectorA(p). The measurement value z(p) in the FIGS. 8a-8c is assumed to beclose to zero (as an example) such that the updated S′(p) have valuesclose to zero where the applicability vector B(p) has high values inspecific wavelengths. As can be seen in FIG. 8b , the updatedapplicability function A′ corresponding to the updated applicabilityvector A′(p) shows at which wavelengths spectral information is added toS(p).

Other update schemes can be used as well, for example a non-linearfunction or a Kalman filter. The applicability vector A(p) is updatedaccordingly.

When the point p has been observed from two different angles (32 and 33in FIG. 4), and it can be established that it is the same point p thatis observed at these angles, the relative 3D position of point p can beestimated by triangulation. For the estimate to have good precision, theangle difference between the two observations should be as large aspossible (up to 90 degrees) and/or the estimate be based on multiple,not only two, observations. Methods for simultaneous estimation ofsensor motion and 3D position of multiple points are known from theliterature.

As mentioned, it must be established that a point 31 observed at by thesensor 10 at one position at one angle 32 is the same point observed bythe sensor 10′ at a second position at a second angle 33. Methods fortracking points through a time sequence of images are known from theliterature—examples are optical flow and KLT-tracking.

Thus, with the present invention, the 3D position of multiple points ona surface (30 in FIGS. 4, 5 a, 5 b), such as the Earth, can beestimated, and at the same time, the amount of light radiated from thesame points can be measured at different wavelengths.

Note that in the present invention, the tracking does not need to becausal. If a buffer of images is kept in the memory of the processingunit (20), tracking and 3D structure computation can be done in eithertemporal direction, or in both directions, or as a batch computation.

A problem that will appear with existing tracking methods is that apoint p and its neighborhood will not appear the same at differentangles, due to the optical filter. For example, a red cross on greenbackground will appear as a white cross on a black background at oneangle (when the cross is projected on a part of the sensor chip wherelight with wavelength corresponding to the color red can pass throughthe filter) and a black cross on a black background at another angle.Such change of appearance will increase the uncertainty of the tracking,which leads to less reliable estimates of 3D structure.

Tracking may be done using an adaptive model. Since the appearance ofthe neighborhood of p will change gradually, a model that updatescontinuously during the tracking process will enhance the accuracy ofthe tracking and thus also the accuracy of the 3D estimate. Adaptivemodels that can be used for such tracking are known from the literature,and can, for example be based on storing an image patch, an imagedescriptor (such as SIFT, SURF or other descriptors known from theliterature), or a representation of a probability distribution field(such as a kernel density estimators, histograms, channel coded vectors,or parameterized distributions).

The tracking may be enhanced by exploiting spectral information. Thatis, a point p of the scene and/or a neighborhood around the point p, isnot only characterized by its appearance, but also by its spectralcharacteristics. This is particularly useful when the projection of apoint p on the FPA is not following a straight line but returns to acolumn (corresponding to a specific wavelength) that has been previouslymeasured. The model matching in the above-mentioned tracking process canthen be improved by adding spectral matching, i.e. comparing the elementof a spectral vector S(p) being the closest correspondence to thewavelength of the hypothetical matching measurement z′(p) and trackingthe point p to the position of the pixel where the hypothetical matchingmeasurement z′(p) is done.

The spectral matching may comprise computing a match between ahypothetical matching measurement z with the current applicabilityvector B(p), the current spectral vector S(p) and its applicabilityvector A(p) as

norm((S(p)−z(p))·*A(p)·*B(p)),

where “.*” denotes element by element multiplication and norm( ) is avector norm, such as the L1 norm or the L2 norm. The resulting matchvalue is combined with the match value from the adaptive model, theexact formulation dependent on the specific adaptive model used.

The adaptive model and the spectral matching may be replaced by anadaptive spatio-spectral model, such as a representation of a 3Ddistribution field with a corresponding 3D applicability field. A 3Ddistribution field can, for example, be represented and estimated usingkernel density estimators, 3D histograms, channel coded 3D fields, orparameterized distributions. Whereas this is not very complicated toimplement in the processing unit, the computational complexity willincrease, and a balance of tracking performance and complexity must befound for optimum performance.

The arrangement of FIG. 9 provides some additional opportunities.

Initially, the use of two or more lenses, which in turn may provide twoor more depictions of the point p on the image sensor, makes it possibleto calculate a distance to the point p. The accuracy of such distancecalculation is dependent on factors, such as the size of the imagesensor, the resolution of the image sensor and the precision of thefilter and other optics involved. Typically, a distance between thesimultaneous depictions of the point on the image sensor may be used asa reference distance.

If the filter 12′ used is of the same kind as the one 12 previouslydiscussed, then the different depictions of the point on the image willbe with different spectral contents. Hence, the methods disclosed abovemay need to be used in order to identify the point p at its differentoccurrences on the same image.

However, the filter 12′ may be of the kind which provides two or moreportions having the same or very similar transmittance. In such anembodiment, the depictions of the point may be with the same, or almostthe same, spectral contents, which may facilitate the identification ofthe point at its occurrences on the image.

3D reconstruction of an observed object, such as a part of the Earth ora building, can also be performed by bundle adjustment, as known fromthe literature. Using such a method, the tracking described above isreplaced by searching for common features in the acquired set of images.Those features can be e.g. hand-crafted features (SIFT, SURF, . . . ) ora learnt features (for example, by neural networks). This process istypically performed as a post-processing step, in contrast to thefeature-tracking approach described above which can be performed online.

Somewhat surprisingly, it has shown that such a search for commonfeatures can be done with accuracy good enough for high-precision 3Dreconstruction, in spite of the variable wavelengths at which thefeatures are imaged. However, to do the hyperspectral reconstruction,some additional steps may be implemented.

The 3D reconstruction method will output three things; the intrinsicparameters of the camera (focal length, lens distortion, etc), a set ofextrinsic parameter sets (camera positions and orientations), up to oneper image acquired, and a set of 3D points. Typically, a 3D surfacepolygon model is also computed. To be able to reconstruct thehyperspectral signature of a point, the 3D coordinates of that point canbe projected onto the image plane of each acquired image using thecorresponding extrinsic and intrinsic parameters. Then, using theresulting image coordinates, a hyperspectral signature can bereconstructed using the interpolation method described above.Additionally, if a surface model is computed, the hyperspectralsignature of an arbitrary point on the surface can be estimated usingthe same interpolation method as above.

This process is extremely time and memory consuming, and thus difficultto use in practice. Thus, we propose a scheme where one or more stepscan be made offline and the remaining be made online.

Below, it is assumed that all the acquired images are stored in adatabase, each image accessible with its index number.

In a first step, for each reconstructed 3D point P, the entire set ofintrinsic and extrinsic camera parameters is traversed. For each match,that is, when the projection of the 3D point to the image plane resultsin valid image coordinates, the index of the image and the imagecoordinates p=(x,y) are stored. Note that the x image coordinateimplicitly tells the wavelength at which the 3D point is measured inthat particular image. Thus, instead of the x coordinate, the centerλ_(c) (x,y) wavelength could be stored. Additionally, instead of the ycoordinate and the image index number, the (interpolated) image value atthe coordinate p could be stored directly. This choice depends on whichof the steps below that should be executed.

As an optional next step, for each reconstructed 3D point, removesuperfluous index-coordinate pairs, that is, such index-coordinate pairs(i,p) where the x coordinate is close to other index-coordinate pairs orwhere x coordinate is outside a pre-defined range. This step is notstrictly necessary, but can significantly speed up the interpolationprocess.

As an optional next step, for each 3D point, collect image patchescentered around each of the collected index-coordinate pairs. Align thisset of image using an image matching technique, such as normalized crosscorrelation, in order to refine the image coordinates.

As the next step, reconstruct the hyperspectral signature of the 3Dpoint. This can be done offline or online (when accessing thatparticular point). The reconstruction is performed by traversing thestored index-coordinate pairs for the 3D point, interpolating the imagevalue at each of these (if not already done in the first step), andfeeding the resulting z to the previously described hyperspectralinterpolation algorithm. Interpolation of the image value can be doneusing standard methods from the literature.

If a surface model has been computed, hyperspectral reconstruction canbe performed at any point on this surface. For such a 3D point P, thefirst step is to identify the surface element to which P belongs.Second, the barycentric coordinates p′ of P in the identified surfaceelement is computed. The barycentric coordinates can then be used asweight on the corresponding hyperspectral signatures and thus computethe hyperspectral signature of P.

1. A method of providing a hyperspectral image of a scene comprising apoint, comprising: providing an imaging device comprising atwo-dimensional image sensing unit having a spectral characteristic,which varies along at least one direction in a plane parallel with animage sensor, acquiring a first two-dimensional image of the scene, thefirst image comprising the point, wherein the spectral content of thescene varies along the direction as a consequence of the varyingspectral characteristic of the image sensing unit, moving the imagingdevice and the scene relative to each other, acquiring a secondtwo-dimensional image of the scene, the second image comprising thepoint, wherein the spectral content of the scene varies along thedirection as a consequence of the varying spectral characteristic of theimage sensing unit, a) identifying the point in the second image as asecond pixel at a position depicting the point, b) providing a spectralvector of the point, c) providing an applicability vector of the pointto describe, with respect to predetermined wavelengths, the spectralcharacteristic of the image sensing unit at a position where the pointis depicted, d) providing an applicability vector of the second pixel,said applicability vector describing, with respect to a predeterminedwavelength, the spectral characteristic of the image sensing unit at theposition of the second pixel, e) providing an updated spectral vector ofthe point based on the applicability vector of the point, the spectralvector of the point, a spectral value of the second pixel and theapplicability vector of the second pixel, f) repeating the above stepsa) to e) for at least one different point of the first two-dimensionalimage of the scene, to provide a respective updated spectral vector forthe point and the different point, and to generate the hyperspectralimage of the scene comprising the respective updated spectral vector forthe point and the different point; wherein the spectral vector of thepoint is provided as an N-dimensional vector comprising a predeterminednumber of wavelengths, wherein N preferably is 5-150. 2-3. (canceled) 4.The method as claimed in claim 1, further comprising providing anupdated applicability vector based on the applicability vector of thepoint and the applicability vector of the second pixel.
 5. The method asclaimed in claim 4, wherein the updated applicability vector is providedas a maximum, on an element by element basis, of the applicabilityvector of the point and the applicability vector of the second pixel. 6.The method as claimed in claim 1, wherein identifying the pointcomprises using the spectral vector of the point.
 7. The method asclaimed in claim 6, wherein the identifying comprises identifying thepoint based on a difference between the second spectral value and thespectral vector.
 8. The method as claimed in claim 6, wherein theidentifying comprises identifying the point based on the spectralvector, the second spectral value, the applicability vector of the pointand the applicability vector of the second pixel.
 9. The method asclaimed in claim 1, wherein identifying the point in the second imagecomprises providing a model of a neighborhood of the point in one of theimages, and matching the model to the one of the images.
 10. The methodas claimed in claim 1, further comprising determining a respectiveobservation angle of the point based on respective positions of thepoint in the first and second images and a reference distance, anddetermining a distance between the imaging device and the point based onthe angles and the reference distance.
 11. The method as claimed inclaim 1, wherein providing a spectral vector of the point comprisesassigning a first spectral value of a first pixel depicting the point inthe first image.
 12. The method as claimed in claim 1, wherein providinga spectral vector of the point comprises receiving the spectral vectoralready updated based on the first image.
 13. The method as claimed inclaim 1, wherein the varying spectral characteristic is provided by avariable optical filter arranged adjacent the image sensor and whereinthe method further comprises providing the applicability vector of thepoint based on a transmittance of the optical filter at each wavelengthat the respective position of the image sensor where the point isidentified. 14-16. (canceled)
 17. The method as claimed in claim 1,wherein a respective spectral vector is provided for a predeterminednumber of points in the scene, said predetermined number of points beingsufficient to positively identify an object or absence of an object inthe scene.
 18. The method as claimed in claim 1, wherein acquiring thefirst and/or second image comprises acquiring an image wherein the pointoccurs only once.
 19. The method as claimed in claim 1, whereinacquiring the first and/or second image comprises acquiring an imagewherein the point occurs at least twice.
 20. The method as claimed in19, further comprising determining a respective observation angle of thepoints based on respective positions of the occurrences of the point inthe first and/or second images and a distance which is an intrinsicdistance of the imaging device.
 21. (canceled)
 22. The method as claimedin claim 1, wherein the method comprises interpolating at least oneelement of the spectral vector, which element corresponds to awavelength for which measurement has been made with respect to thepoint, preferably, the estimation is performed based on theapplicability vector.
 23. The method as claimed in claim 1, furthercomprising: if the point has moved more than one column on the imagesensing unit between the first and second images, then estimating,preferably by interpolating, elements of the spectral vectorcorresponding to wavelengths where no measurement has been done,preferably, the estimation is performed based on the applicabilityvector.
 24. (canceled)
 25. A method of providing a hyperspectral 3Drepresentation of an object, comprising: providing a plurality ofspectrally varying images depicting the object, providing a 3Dreconstruction of the object based on the plurality of spectrallyvarying images, and assigning a hyperspectral signature to a pluralityof 3D points on a surface of the object. 26-32. (canceled)
 33. Animaging device, comprising: a two-dimensional image sensor, presentingfirst and second mutually orthogonal directions, an optical filter,arranged in an optical path to the image sensor, wherein the opticalfilter is an optical variable bandpass filter, the transmittance ofwhich varies along on the first direction and being constant along thesecond direction, wherein the optical filter extends over at least 90%,preferably 95%, or preferably 99% of a length of the image sensor in thefirst and second directions, respectively, and wherein the imagingdevice is arranged to acquire two-dimensional images of a scene, whereinthe spectral content of the scene as depicted on the images varies alongthe first direction as a consequence of the varying transmittance of theoptical filter. 34-37. (canceled)
 38. The imaging device as claimed inclaim 33, further comprising a processing device, which is configured toperform the method as claimed in claim 1.